Naval  Aerospace  Medical  Research  Laboratory 


NAMRL-1395 

THE  PREDICTION  ALGORITHM  FOR 
THE  LANDING  CRAFT  AIR  CUSHION 
(LCAC)  SELECTION  SYSTEM 

D.  J.  Blower 


199T0203  094 


NaVal  Aerospace  Medical  Research  Laboratory 
51  Hovey  Road  ^  ^ 

Pensacola,  Florida  32508-1046 


Approved  for  public  release;  distribution  unlimited. 


Reviewed  and  approved  \  V)0 0 


L.  H.  FRANK,  CAPT,  MSC  USN 
Commanding  Officer 


This  research  was  sponsored  by  the  Naval  Sea  Systems  Command  (Code  PMS  377)  through  the  Naval  Ocean 
Systems  Center  and  supported  by  the  Naval  Coastal  Systems  Center  under  work  unit  number  CE02. 

The  views  expressed  in  this  article  are  those  of  the  authors  and  do  not  reflect  the  official  policy  or  position  of  the 
Department  of  the  Navy,  Department  of  Defense,  nor  the  U.S.  Government. 

Trade  names  of  materials  and/or  products  of  commercial  or  nongovernment  organizations  are  cited  as  needed  for 
precision.  These  citations  do  not  constitute  official  endorsement  or  approval  of  the  use  of  such  commercial  matenals 

and/or  products. 

Reproduction  in  whole  or  in  part  is  permitted  for  any  purpose  of  the  United  States  Government. 


NAVAL  AEROSPACE  MEDICAL  RESEARCH  LABORATORY 
51  HOVEY  ROAD,  PENSACOLA,  FL  32508-1046 


NAMRL-1395 

THE  PREDICTION  ALGORITHM  FOR  THE  LANDING  CRAFT  AIR 
CUSHION  (LCAC)  SELECTION  SYSTEM 


D.  J.  Blower 


Approved  for  public  release;  distribution  unlimited. 


Abstract 


This  paper  explains  the  prediction  algorithm  used  by  the  Landing  Craft  Air  Cushion  Vehicle 
(LCAC)  selection  system.  Five  variables  from  a  psychomotor  test  battery  were  combined  to  form 
a  composite  score.  This  composite  score  was  then  compared  to  a  threshold  score.  If  a  candidate 
for  an  LCAC  crew  position  achieved  a  composite  score  higher  than  the  threshold  score,  that 
candidate  was  predicted  to  pass  Phase  I  of  LCAC  training.  Likewise,  if  a  candidate  scored  lower 
than  the  threshold  score  he  was  predicted  to  fail  training.  The  threshold  score  was  determined 
by  Statistical  Decision  Theory  as  interpreted  from  the  Bayesian  approach.  Examples  are  given 
showing  how  the  threshold  scores  can  change  as  a  function  of  the  prior  probabilities  of  pass  or 
fail  and  the  values  attached  to  making  correct  and  incorrect  predictions. 
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1  Introduction 


The  Naval  Aerospace  Medical  Research  Laboratory  (NAMRL)  first  began  research  into  a  selection 
system  for  the  Landing  Craft  Air  Cushion  (LCAC)  Vehicle  in  1987  with  funding  from  the  Naval 
Sea  Systems  Command.  At  that  time,  the  attrition  rate  during  Phase  I  training  for  the  LCAC 
operator  position  was  approximately  40%.  Since  NAMRL  had  been  involved  in  aviator  selection, 
it  was  thought  that  some  of  the  same  tests  might  help  to  reduce  these  high  LCAC  attrition  rates. 
Even  thou^  the  NAMRL  selection  test  battery  had  been  devised  for  officers  in  the  aviation 
community,  the  tasks  involved  in  piloting  the  LCAC  might  exhibit  enough  similarity  to  use  them 
for  the  enlisted  group  targeted  as  LCAC  operators. 

Accordingly,  the  NAMRL  selection  test  battery  was  modified  to  form  an  LCAC  selection 
system,  and  research  was  begun  to  assess  its  efficacy  in  reducing  attrition  for  the  LCAC  operator 
community.  The  selection  system  proved  so  successful  that  it  was  enlarged  to  encompass  two 
other  LCAC  positions,  the  engineer  and  the  navigator.  The  LCAC  selection  system  is  now 
routinely  administered  to  about  100  candidates  yearly  for  the  operator  and  engineering  positions 
at  the  Naval  Aerospace  and  Operational  Medical  Institute.  Modifications  to  the  LCAC  selection 
system  to  accommodate  some  unique  requirements  for  the  navigator  position  are  currently  being 
researched  at  NAMRL.  The  intent  of  this  rqjort  is  to  document  the  prediction  algorithm  at  the 
heart  of  the  LCAC  selection  system.  There  are  many  other  references  [1-15]  that  describe  the 
history  of  the  research  and  the  implementational  details. 

The  LCAC  selection  system  contains  an  algorithm  for  predicting  the  success  or  failure  of  an 
individual  candidate  who  might  possibly  undergo  training  for  the  position  of  operator  or  engineer. 
The  actual  implementation  of  the  algorithm  in  the  software  code  is  quite  simple.  Five  variables 
from  a  psychomotor  test  battery  are  weighted  and  then  combined  to  form  a  composite  score. 
This  composite  score  is  assumed  to  be  a  good  indicator  of  the  more  complex  psychomotor  skills 
relevant  to  success  during  LCAC  training.  Each  candidate’s  composite  score  on  the  test  battery 
is  then  compared  to  a  threshold  score.  If  a  candidate’s  composite  score  is  greater  than  or  equal 
to  the  threshold  score,  the  candidate  is  predicted  to  pass  training,  while  a  composite  score  below 
the  threshold  results  in  a  prediction  of  failure  for  the  candidate. 

How  were  the  variables  and  the  associated  weights  that  make  up  the  composite  score  derived? 
Briefly,  scores  were  constructed  such  that  the  distance  between  the  mean  score  of  those  who 
eventually  passed  training  and  the  mean  score  of  those  who  eventually  failed  training  was  as 
far  apart  as  possible.  At  the  same  time,  the  variance  of  the  composite  scores  was  kept  to  a 
minimum  for  each  of  these  two  groups.  This  goal  was  accomplished  via  the  statistical  technique 
of  Discriminant  Analysis. 

While  the  actual  implementation  of  the  algorithm  in  the  software  code  is  relatively  straightfor¬ 
ward,  the  mathematical  rationale  for  where  to  set  the  threshold  that  predicts  the  training  outcome 
is  a  bit  more  complex.  The  development  depends  upon  a  branch  of  mathematics  called  statistical 
decision  theory  (SDT).  SDT,  in  turn,  hinges  on  a  Bayesian  treatment  for  making  inferences  when 
costs  and  prior  probabilities  can  be  specified. 

If  the  costs  of  the  correct  and  incorrect  decisions  and  prior  probabilities  for  the  outcomes  can 
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be  specified,  then  SDT  will  provide  the  optimal  placement  of  the  threshold  based  on  this  input. 
Since  these  subjective  and  economic  data  are  generally  not  disclosed  to  researchers  involved  in 
selection,  it  is  left  up  to  the  developers  of  the  LCAC  selection  system  to  interpret  the  desires  of  the 
LCAC  community  in  making  a  determination  of  the  optimal  pass/fail  threshold  score.  Arbifrarily 
choosing  a  threshold  that  is  high  will  undoubtedly  rule  out  the  selection  of  a  certain  number  of 
candidates  that  would  otherwise  be  successful  in  training;  however,  if  the  costs  associated  vwth 
candidate  failure  during  training  are  high,  and  outweigh  the  costs  associated  with  eliimnahng 
some  potentially  successful  candidates,  then  such  a  high  pass/fail  threshold  rmght  be  desirable. 

Even  a  successful  selection  test  battery  and  associated  prediction  algorithm  like  the  LCAC 
selection  system  cannot  predict  with  complete  certainty  whether  a  candidate  will  pass  or  fail 
training.  It  should  be  understood  that  the  LCAC  selection  system  raises  Hiq  probability  for  any 
candidate  selected  to  successfully  complete  training.  For  example,  before  Ae  selection  system  was 
implemented,  the  consequent  random  sampling  of  psychomotor  skills  might  result  in  an  overal 
probability  of  success  equal  to  .50.  With  a  selection  system,  by  sampling  some  of  the  relevant 
psychomotor  skills  to  successfully  operate  the  LCAC,  this  probability  might  be  raised  to  .80.  This 
means  that  there  will  be  less  attrition  during  training,  but  does  not  mean  that  every  candidate  who 
passes  the  test  battery  will  pass  training.  Any  candidate  chosen  by  the  test  battery  has  a  higher 
probability  of  completing  training  than  one  chosen  without  the  benefit  of  this  information. 

Any  selection  test  that  is  not  perfectly  correlated  with  training  outcome  (and  this  includes  all 
selection  tests)  will  exhibit  some  recommendations  that  will  later  turn  out  to  be  in  error.  The 
correctness  of  decisions  in  trainee  selection  is  related  to  the  degree  to  which  the  selection  test  taps 
into  the  skills  needed  to  perform  critical  tasks.  For  example,  perhaps  docking  the  LCAC  aboard 
the  mothership  is  a  skill  that  is  not  sampled  by  the  selection  test.  A  candidate  for  traimng  then 
can  perform  adequately  on  the  psychomotor  test  battery  and  yet  fail  the  training  where  this  skill  is 
needed.  There  are  also  those  additional  failures  during  training  that  the  selection  test  could  never 
have  predicted  because  they  are  due  to  extraneous  factors  such  as  morale,  motivation,  unique 
situational  occurrences,  family  concerns,  personal  life  style  changes,  etc. 

For  all  of  these  reasons,  there  will  always  be  failures  from  the  LCAC  training  program  even 
after  the  selection  test  battery  has  given  a  stamp  of  approval.  Raising  the  threshold  can  help  reduce 
failures,  but  it  is  not  a  universal  panacea.  There  comes  a  point  where  raising  the  threshold,  even 
though  it  does  provide  you  with  a  candidate  with  better  psychomotor  skills,  cannot  compensate 
for  those  other  factors,  mentioned  above,  that  can  cause  failure.  This  is,  of  course,  just  anoAer 
way  of  saying  Aat  Ae  selection  test  battery  cannot  predict  success  or  failure  with  a  probability 

of  1. 

AnoAer  consideration  to  bear  in  mind  in  connection  wiA  Ae  probabilistic  naAre  of  Ae  LCAC 
selection  system  is  Aat  even  Aough  we  talk  about  raising  Ae  probability  of  success  in  training 
from,  say,  .50  to  .80,  Ae  acAal  numbers  passing  or  failing  during  traimng  over  a  fimte  s^ple 
size  ^11  vary  from  Ae  expectation  dictat^  by  Aese  probabilities.  Over  a  finite  sample  size  of 
candidates,  say  100  for  an  easy  number,  50  will  be  expected  to  pass  and  50  will  be  expected  to 
fml  wiAout  Ae  selection  test.  But  it  could  happen  through  random  flucAations  over  these  100 
canAdates  Aat  only  40  pass  or  perhaps  60  pass,  while  Ae  underlying  probability  has  not  deviated 
from  .5.  Likewise,  if  Ae  underlying  probability  has  been  raised  Arough  the  use  of  the  selection 
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system  to  .8,  we  would  expect  80  to  pass,  but  would  not  be  surprised  if  only  72  passed  or  86 
passed.  In  both  cases,  these  numbers  are  compatible  with  a  heightened  probability  of  success  of 
.8  and  reflect  only  the  vagaries  of  the  sample  size.  However,  an  extra  benefit  of  the  increased 
probability  that  use  of  the  selection  system  affords  is  that,  although  sampling  variability  still 
exists,  it  is  smaller  at  higher  probabilities  than  at  .5. 


2  The  Derivation  of  the  Composite  Scores 

Five  variables  were  chosen  from  the  psychomotor  test  battery  through  discriminant  analysis. 
These  five  variables  were  reduced  to  four  by  combining  the  last  two  variables.  Further  research 
may  indicate  other  variables  that  could  be  included  in  a  revised  prediction  algorithm,  but  at  the 
present  time  these  variables  seem  to  perform  well.  The  composite  score  (CS)  was  derived  by 
multiplying  these  four  variables  by  weights  suggested  by  the  discriminant  analysis  and  adding  a 
constant  value.  The  composite  scores  used  in  the  numerical  examples  for  a  candidate  who  was 
a  predicted  PASS  and  for  the  candidate  who  was  a  predicted  FAIL  are  derived  in  the  two  tables 
shown  below.  The  actual  formula  for  the  composite  score  can  be  expressed  as  follows, 

4 

CS  =  Y.'^iSi  +  c 

i=l 

=  (—1)5^1  +  (— 4)jS2  +  (—1)63  +  (+1)54  +  27 
=  27  +  Si-Si-AS2-Sz 

As  will  be  explained  in  the  next  section,  the  threshold  composite  score  was  calculated  as  +.14.  If 
a  candidate’s  performance  on  the  test  battery  as  reflected  by  the  composite  score  is  above  +.14, 
the  candidate  is  a  predicted  PASS.  If  a  candidate’s  performance  on  the  test  battery  is  below  +.14, 
the  candidate  is  a  predicted  FAIL.  The  table  below  shows  a  candidate  who  scored  fairly  well  on 
the  test  battery.  Since  the  composite  score  of  +2.00  is  above  +0.14,  this  candidate  is  a  predicted 
PASS. 

The  variables,  a  typical  value  for  each  variable,  their  associated  weights,  and  the  resulting 
composite  score  are  presented  in  Table  1  below. 


Variable 

Name 

Value 

Weight 

Multiplication 

Stick  and  DLT 

4.00 

-1 

-4.00 

Stick  and  Rudder 

52 

4.50 

-4 

-18.00 

Stick,  Rudder,  and  Throttle 

53 

4.75 

-1 

-4.75 

Horizontal  Tracking  and  RT 

54 

1.75 

+1 

1.75 

Constant 

27.00 

+1 

27.00 

Composite  Score 

+2.00 
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The  tracking  error  scores  shown  in  the  first  three  rows  of  this  table  represent  a  logarithmic 
transformation  of  the  actual  raw  pixel  error  in  order  to  promote  a  more  normal  distribution  for  this 
kind  of  score.  The  fourth  variable  is  itself  a  composite  variable  consisting  of  a  denved  measure 
of  tracking  while  performing  a  digit  cancellation  task  at  the  same  time.  LCAC  candidate 
with  this  kind  of  composite  score  would  have  a  calculated  posterior  probability  for  PASS  greater 
than  ,50  and  therefore  would  have  been  a  predicted  PASS  using  the  prediction  algorithm.  TWs 
statement  is  explained  in  subsequent  sections  dealing  with  the  Bayesian  nature  of  the  prediction 
algorithm. 

Table  2  shows  a  candidate  who  did  not  do  as  well  on  the  test  battery.  Notice  that  Si,  ^2,  and 
S3  are  error  scores  so  that  a  higher  score  means  worse  performance.  Lower  scores  on  variable 
<S'4  we  associated  with  worse  performance  so  that  this  subject  also  did  not  perform  as  well  as 
the  previous  subject  on  Horizontal  Tracking  and  RT.  This  subject  achieved  a  composite  score  of 
-1.50  and  is,  therefore,  a  predicted  FAIL  since  this  composite  score  is  below  the  threshold  score. 


Variable 

Name 

Value 

Weight 

Multiplication 

Stick  and  DLT 

4.50 

-1 

-4.50 

Stick  and  Rudder 

52 

5.00 

-4 

-20.00 

Stick,  Rudder,  and  Throttle 

53 

4.90 

-1 

-4.90 

Horizontal  Tracking  and  RT 

Si 

.90 

+  1 

.90 

Constant 

27.00 

+1 

27.00 

Composite  Score 

-1.50 

3  Bayes’s  Formula  and  Numerical  Examples 


The  prediction  algorithm  for  deciding  whether  to  recommend  a  candidate  for  entrance  into  the 
LCAC  training  program  is  based  on  the  composite  score  from  the  psychomotor  test  battery  as 
just  explained.  However,  this  is  just  part  of  the  story  behind  the  prediction  algorithrn.  The  use  of 
statistical  decision  theory,  together  with  the  assignment  of  relative  costs  to  a  decision-true  state 
matrix,  lies  at  the  core  of  statistical  decision  theory  as  it  is  used  here.  This  Bayesian  approach 
seeks  to  update  a  prior  state  of  knowledge  about  the  probability  of  su^ess  in  LCAC  training 
through  new  information  concerning  skills  thought  to  be  relevant  for  training  success. 

An  LCAC  candidate  is  classified  as  a  predicted  PASS  or  FAIL  by  computing  the  posterior 
probability  of  belonging  to  the  PASS  or  FAIL  group,  and  then  assigning  the  candidate  to  the  ^oup 
with  the  largest  posterior  probability.  The  extra  information  used  to  update  the  prior  probabilities 
is  the  composite  score  derived  from  the  psychomotor  test  battery.  The  general  form  of  Bayes  s 
Formula  used  to  calculate  the  posterior  probability  of  group  membership  is 


P{Gi\D)  = 


P{D\Gi)  X  P{Gi) 
j:i=iPiD\Gi)xP{Gi) 


(1) 


where  P{Gi\D)  is  the  posterior  probability  for  Group  i  after  receipt  of  some  test  data,  P{D\Gi) 
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is  the  likelihood  of  the  test  data  ^ven  Group  i,  and  P{Gi)  is  the  prior  probability  of  Group  i. 

In  particular,  for  the  two  group  case,  which  is  the  focus  of  our  attention  in  the  LCAC  study, 
the  formula  reduces  explicitly  to 


P(PASS|i)) 


P(D|PASS)  X  P(PASS) 

[PpiPASS)  X  P(PASS)]  +  [P(D|FAIL)  x  P(FAIL)] 


(2) 


P(FAILID)  =  1-P(PASS|P) 


(3) 


The  likelihood  terms, 
are  Gaussian, 


P(D|PASS)  and  P(I>|FAIL) 


P(I>|PASS)  = 


1 


y/^(^PAss 


\2 

X-tlpASS  \ 
<^PASS  ) 


P(I»|FAIL)  = 


1 


yP^^FAlL 


e 


x-hfail\ 

<^FAIL  J 


2 


(4) 

(5) 


where  x  is  the  candidate’s  composite  score,  fipAss  is  the  group  mean  of  the  composite  scores 
for  the  PASS  group,  ixfail  is  the  group  mean  of  the  composite  scores  for  the  FAIL  group,  and 
(^PAss  and  (Tfail  are  the  standard  deviations  of  the  computed  composite  scores. 

The  discriminant  analysis  (DA)  program  from  the  statistical  software  package  SPSSPC+  was 
used  to  to  derive  an  appropriate  composite  score.  The  DA  program  calculated  /j^pass  =  .77239 
and  fiFAiL  =  .14135.  apAss  and  (tfail  are  both  equal  to  1  by  definition.  Equation  (2)  can  then 
be  written  as. 


P(PASSID)  = 

fi 

fl  +  /2 

(6) 

where  /i  = 

exp{-l/2  (a;  -  fipAss)^}  x  P(PASS) 

(7) 

and/2  = 

exp{-l/2  {x  -  (j-fail)^}  x  P(FAIL) 

(8) 

A  numerical  example  using  this  formula  for  the  calculation  of  the  posterior  probability  of 
belonging  to  the  PASS  or  FAIL  group  follows.  Assume  that  a  candidate  takes  the  selection  test 
battery  and  does  reasonably  well  on  the  test.  His  composite  score  turns  out  to  be  equal  to  2.00 
just  as  in  the  example  of  the  previous  section.  A  summary  of  the  variables  needed  to  solve 
Equations  (6) — (8)  is  presented  in  Table  3.  The  reason  for  these  particular  values  of  the  prior 
probabilities  will  be  explained  later. 
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The  composite  score  for  the  LCAC  candidate 

a;  =  2.00 

Group  mean  of  composite  scores  for  PASS 

fipAss  =  -77239 

Group  mean  of  composite  scores  for  FAIL 

fJ’FAIL  =  .14135 

Prior  probability  of  PASS 

P(PASS)  =  .53 

Prior  probability  of  FAIL 

P(FAIL)  =  .47 

/i  =  exp{-l/2  (2.00 77239)2}  X. 53 
=  .4707  X. 53 

/s  =  exp{-l/2  (2.00 14135)2}  X. 47 


=  .1778  X. 47 

.4707  X  .53 

P(PASS|£>  =  2.00)  =  ^  ^  ^ 

.2495 

.2495  +  .0835 
=  .7491 

Therefore,  the  posterior  probability  of  a  PASS  in  LCAC  trdning  is  .7491  given  that  a  c^didate 
achieved  a  composite  score  of  2.00.  It  follows  that  the  posterior  probability  of  a  FAIL  is  .2509. 
The  probability  of  belonging  to  the  PASS  group,  given  the  data  from  the  test  battery,  is  about 
75%.  This  candidate  will  be  classified  by  the  system  as  a  PASS. 

Another  example  of  this  formula  is  presented  below;  this  time  of  an  LCAC  caiididate  with 
poorer  performance  on  the  test  battery.  Whereas  in  the  first  example  the  LCAC  candidate  scored 
a  2.00  and  was  classified  as  a  PASS,  in  this  example  the  candidate  scores  a  -1.50  as  a  composite 
score.  What  is  this  candidate’s  predicted  training  outcome  based  on  these  data?  As  in  the 
previous  case,  the  relevant  data  are  presented  in  Table  4. 


The  composite  score  for  the  LCAC  candidate 

a;  =  -1.50 

Group  mean  of  composite  scores  for  PASS 

jJ'PASs  =  .77239 

Group  mean  of  composite  scores  for  FAIL 

IJ-FAIL  =  .14135 

Prior  probability  of  PASS 

P(PASS)  =  .53 

Prior  probability  of  FAIL 

P(FAIL)  =  .47 

6 


/i  =  exp{-l/2  (-1.50 77239)2}  X  .53 
=  .0756  X. 53 


/a  =  exp{-l/2  (-1.50 -.14135)2}  X  .47 


P{VASS\D  =  -1.50) 


.2600  X  .47 

.0756  X  .53 

(.0756  X  .53)  +  (.2600  x  .47) 
.0401 

.0401  +  .1222 
.2470 


Therefore  the  posterior  probability  of  a  PASS  in  LCAC  training,  given  that  a  candidate  achieved 
a  composite  score  of  -1.50,  is  .2470.  Again,  it  follows  that  the  posterior  probability  of  a  FAIL 
for  such  a  score  is  .7530.  The  posterior  probability  of  belon^ng  to  the  FAIL  group  is  higher  than 
the  posterior  probability  of  belonging  to  the  PASS  group,  so  this  candidate  is  predicted  to  FAIL. 

We  do  not  have  to  actually  cany  out  these  calculations  to  make  a  prediction  for  a  candidate. 
We  simply  have  to  find  out  beforehand  the  threshold  score,  i.e.,  the  score  such  that  anything  above 
this  score  is  a  predicted  PASS  and  anything  below  it  is  a  predicted  FAIL.  Then  the  algorithm 
can  make  a  prediction  by  simply  comparing  a  candidate’s  composite  score  with  the  threshold 
score.  The  threshold  score  is  that  composite  score  where  a  predicted  PASS  and  a  predicted  FAIL, 
based  on  the  data  from  the  test  battery,  are  both  equal  to  .50.  With  the  prior  probabilities  set 
at  P(PASS)  =  .53  and  P(FAIL)  =  .47  in  the  above  numerical  examples,  this  threshold  score  is 
.14.  This  is  observed  to  be  the  mean  of  the  composite  scores  for  the  FAIL  group.  As  a  matter  of 
fact,  the  prior  probabilities  were  purposely  manipulated  so  that  the  threshold  score  would  be  at 
this  precise  location. 

Different  prior  probabilities  would  dictate  a  different  threshold  score  where 

P(PASS|Data)  =  P(FAIL|Data)  =  .50 

In  the  Bayesian  approach  to  statistical  decision  theory,  the  threshold  score  is  determined  by  a 
multiplicative  relationship  between  the  costs  associated  with  the  available  decisions  and  the  prior 
probability.  However,  the  DA  program  was  coded  in  such  a  way  that  it  was  possible  to  enter 
only  the  prior  probability,  so  it  has  to  assume  the  entire  burden  for  both  of  these  factors.  When 
a  prior  probability  is  specified,  for  example,  P(PASS)  =  .53  and  P(FAIL)  =  .47,  this  reflects 
both  the  historical  frequency  of  passing  and  failing  as  well  as  the  matrix  of  costs  associated  with 
the  decisions. 

After  taking  into  consideration  both  the  historical  frequency  of  passing  and  failing  and  the 
costs  associated  with  the  predictions  and  the  actual  training  outcomes,  the  threshold  score  was 
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set  at  the  mean  of  the  FAIL  group.  This  decision,  in  turn,  mandated  the  particular  values  for 
the  prior  probability  parameters  of  the  DA  program  that  were  used  in  the  examples.  Different 
historical  frequencies  and  a  different  cost  matrix  would  result  in  different  numerical  settings  for 
these  parameters.  These  different  settings  might  reflect  either  a  change  in  attrition  rates  or  a 
desire  to  restrict  or  promote  the  flow  of  students  into  the  training  pipeline. 


4  The  Evaluation  of  Correct  and  Incorrect  Decisions 

In  statistical  decision  theory,  it  is  not  enough  to  just  provide  the  posterior  probabilities  for  a 
candidate  to  pass  or  fail  training.  There  must  be  some  means  for  takiiig  action  based  on  such 
probabilities.  Within  the  Bayesian  framework,  an  incentive  for  taking  action  is  cast  quantitatively 
by  assigning  values  to  the  matrix  of  all  possible  decisions  with  all  possible  true  states  of  the  world. 

The  construction  of  such  a  matrix  is  quite  feasible  in  our  problem  because  there  are  only  two 
alternatives  to  be  paired  off  with  two  possible  decisions.  The  matrix  will  then  be  a  2  x  2  matrix 
with  four  cells.  The  two  possible  alternatives  represent  the  true  state  of  nature,  i.e.  ,  whether  the 
candidate  actually  passes  or  fails  training.  The  two  possible  decisions  are  to  predict  a  PASS  or 
to  predict  a  FAIL. 

To  complete  the  decision  process,  it  is  necessary  to  fill  in  these  four  cells  with  numbers 
that  represent  the  relative  value  we  ascribe  to  the  two  correct  predictions  and  the  two  incorrect 
predictions.  The  convention  employed  in  this  report  is  to  use  positive  numbers  for  the  correct 
decisions  and  zero  for  the  incorrect  decisions.  See  Fig.  1  for  such  a  2  x  2  matrix  with  values 
specified  generically  as  Vi  through  14- 


Pass 

TRUE 

Fail 


PREDICTION 
Pass  Fail 


Correct 

Incorrect 

Vs 

Incorrect 

Vs 

Correct 

4 

Figure  1;  The  value  matrix  for  the  LCAC  prediction  algorithm.  There  are  only  two  possible  true  states  of 
the  world  and  only  two  possible  decisions.  The  two  correct  and  the  two  incorrect  decisions  are  labelled. 

The  formula  for  reaching  a  decision  about  any  individual  candidate  is  now  derived.  In  the 
literature,  this  is  sometimes  called  “setting  a  response  threshold.”  The  main  concept  from  statistics 
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that  is  used  in  the  following  derivation  is  the  definition  of  expectation,  or  more  informally,  the 
average.  For  a  discrete  variable,  the  expectation  is  defined  as, 

E{X)  =  j^PiXi  (9) 

t=i 

where  the  Pi  are  probabilities  for  each  Xi,  and  the  Xi  are  the  values  of  the  variable  we  are 
interested  in.  In  our  application,  the  pi  are  posterior  probabilities  of  passing  or  failing  based 
upon  data,  and  the  Xi  are  the  values  assigned  to  the  2  x  2  matrix  that  reflect  our  evaluation  of 
the  predictions  and  true  outcomes. 

The  goal  is  to  find  the  expected  value  of  a  PASS  and  compare  it  to  the  expected  value  of  a 
FAIL.  If  the  expected  value  of  a  PASS  is  greater  than  the  expected  value  of  a  FAIL,  then  this 
provides  the  justification  for  acting  and  making  a  prediction  of  a  PASS  for  a  candidate.  We  only 
have  to  average  over  two  Xi  and  two  pi  to  calculate  the  expected  value  of  a  PASS  or  the  expected 
value  of  a  FAIL.  This  is  a  simple  application  of  the  general  rule  of  Bayesian  decision  making 
that  calls  for  the  minimization  of  the  expectation  of  the  loss  function  with  respect  to  the  posterior 
probability  (Smith  [16]). 

4.1  Mathematical  Derivation  of  Threshold  Score 

This  derivation  follows  closely  the  one  given  by  Coombs,  Dawes  and  Tversky  [17].  Let 

Pi  =  P  (Pass  I  Data) 

P2  =  P(Fail|Data) 

^^^(Pass)  =  (li  X  pi)  +  (Va  X  P2) 
jE;F(Fail)  =  (V2xpi)  +  (l4xp2) 

If  we  let  the  expected  value  of  a  PASS  be  greater  than  the  expected  value  of  a  FAIL  in  order  to  reach  a 
decision  about  predicting  a  pass  then, 

PF(Pass)  >  PF(Fail) 

(Fi  X  pi)  +  (V^  X  P2)  >  {V2  X  pi)  +  (F4  X  P2) 

(Vi  xpi)  -  (F2  xpi)  >  (F4  X  P2)  -  (Fa  X  P2) 

Pi{Vi-V2)  >  P2{Va-Vz) 

P_i  >  F4-F3 

P2  “  V1-V2 

P(Pass|Data)  ^  ^4-^3 

P(Fail|Data)  “  Fi  -  F2 


Since  the  left-hand  side  of  the  final  equation  in  the  above  derivation  is  the  ratio  of  posterior  probabiUties, 
we  plug  in  the  likelihood  times  prior  probability  relationship  fiom  Bayes’s  Fomula  to  find, 

F(Pass|Data)  _  P(Data|Pass)  x  P(Pass) 

F(Faa|Data)  ”  F(Data|Fail)  x  F(Fail) 

The  denominator,  F(Data),  cancels  out  in  forming  the  ratio.  Substituting  this  relationship  between  the 
likelihood  and  the  prior  probability  in  the  derivation  of  the  ejqpected  value  gives  us. 


F(Data|Pass)  x  P(Pass)  ^  V4  -  Vs 
P(Data|Fail)  x  P(Fail)  -  V1-V2 

At  this  point,  we  want  to  isolate  the  likelihood  ratio  on  the  left  side  of  this  inequality  so  the  ratio  of  prior 
probabilities  is  moved  to  the  ri^t  side  resulting  in, 

P(Data|Pass)  ^  P(Fail)  ^  V4-V3 
P(Data|Fail)  ~  P(Pass)  14  -  V2 

Notice  that  in  transferring  the  ratio  of  prior  probabilities  from  the  left-hand  side  of  the  equation  to  the 
right-hand  side,  the  ratio  is  inverted.  These  expressions  on  both  sides  of  die  equation  have  traditionally 
been  labelled  in  the  following  manner;  The  expression  on  the  left-hand  side,  the  likelihood  ratio,  is  given 
the  notation. 


r(  \  - 

~  P(Data|Fail) 


(10) 


while  the  expression  on  the  right-hand  side,  consisting  of  the  prior  probabilities  and  the  evaluation  of  the 
correct  and  incorrect  decisions,  is  given  the  notation  of  P, 


P(Fail)  14-14 
^  P(Pass)  14—14 


(11) 


If  C{x)  >  P  then  EV{?ass)  >  EV (Fail)  (12) 

So,  in  this  condensed  notation,  the  decision  process  is  easily  written  down  as  summarized  in  Table  5. 


Predict  PASS 

C{x)  >  p 

Predict  FAIL 

C{x)  <  p 

Data  from  test  battery 

X 

Threshold  score 

P 
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4.2  Numerical  examples  of  setting  the  threshold  score 

Equation  (12)  shows  that  the  score  that  divides  the  predicted  PASS  from  the  predicted  FAIL  is  called  the 
threshold  score.  The  threshold  score,  13,  is  seen  from  Equation  (11)  to  be  a  function  of  two  components, 
1)  the  prior  probabilities  and  2)  the  values  given  to  the  correct  and  incorrect  decisions.  The  first  example 
illustrates  the  threshold  calculation  when  the  prior  probabilities  are  equal  and  the  values  given  to  file  correct 
and  incorrect  decisions  are  also  equal.  If  the  prior  probabilities  for  only  two  possibilities  are  to  be  equal, 
they  must  equal  1/2.  The  values  associated  with  fiie  correct  decisions  are  14  and  14-  They  are  given 
values  of  1.  The  values  associated  with  the  incorrect  decisions  are  14  and  14-  They  are  given  values  of  0. 


P(Fail)  =  1/2 
P(Pass)  =  1/2 


Fi  =1 
14  =0 
Vs  =0 

14  =1 

^  P(Fail)  ^4-14 

^  P(Pass)  14  -  V2 


1-0 

1-0 


The  threshold  score,  or  cut-off  score,  is  therefore  set  at  1.  So  whenever  jC,{x)  >  1,  the  student  is  predicted 
to  pass,  and  whenever  £(x)  <  1,  the  student  is  predicted  to  foil.  This  placement  of  fiie  threshold  score 
results  from  making  rational  decisions  about  average  cost  with  probabilities  of  cost  determined  by  the 
posterior  probabilities. 

Figure  2  shows  that  the  threshold  score  is  located  exactly  where  the  distribution  of  test  scores  for  the 
PASS  and  FAIT,  groups  intersect.  Of  course,  this  intersection  point  is  where, 

P(Data|Pass)  =  P(Data|Fail) 


or  equivalently  where. 


C{x) 


P(Data|Pass) 

P(Data|Fail) 

1 


Therefore,  any  score  to  the  right  of  this  threshold  score  is  predicted  to  pass,  and  any  score  to  the  left  is 
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Fail  Pass 


Figure  2:  The  threshold  score  is  located  where  the  likelihood  ratio  is  equal  to  1. 
predicted  to  feil. 

In  this  next  example,  we  change  one  of  the  two  components  that  make  up  /?  to  observe  what  effect  this 
has  on  shifting  the  threshold  score.  We  choose  to  manipulate  the  prior  probability  by  increasing  the  prior 
probability  of  a  FAIL  from  its  previous  setting  at  1/2,  but  maintain  the  same  value  of  the  costs  associated 
with  the  decisions. 


P(Fail) 

=  .80 

P(Pass) 

=  .20 

=  1 

V2 

=  0 

V3 

=  0 

=  1 

P(Fail)  V4-V3 
~  P(Pass)  ^  V1-V2 

.8  1-0 
.2^1-0 

=  4 
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la  this  case,  the  threshold  score  is  moved  along  the  axis  to  the  right.  It  must  move  to  that  point  where  the 
ratio  of  the  densities, 

P(Data|Pass)  _  . 

P(Data|Fail) 

This  movement  of  the  threshold  score  when  the  prior  probabihty  is  changed  is  shown  schematically  in 
Fig.  3.  Because  of  the  hi^er  prior  probability  of  a  FAIL,  irrespective  of  what  the  test  data  show,  a  student 
must  score  higher  to  move  into  the  region  where  a  PASS  can  be  predicted. 


Fail  Pass 


Figure  3:  The  threshold  score  has  moved  to  the  right  because  the  prior  probabilities  have  changed  to 
favor  a  prediction  of  fail. 

If  the  situation  were  reversed  so  that  the  prior  probability  of  a  PASS  were  .80,  then  the  threshold 
would  move  in  the  opposite  direction  to  where  C{x)  =  1/4.  Now  the  threshold  score  is  lower,  reflecting 
the  symmetry  of  flie  movement  in  response  to  changes  in  the  prior  probability.  If  there  is  knowledge, 
independent  of  the  score  on  the  test  battery,  that  a  student  will  more  likely  pass  than  feil,  this  knowledge 
is  c^tured  in  the  prior  probability.  These  last  two  examples  illustrate  that  this  knowledge  has  affected  the 
threshold  score  placement  in  a  logical  fashion. 

Having  manipulated  die  first  component,  the  ratio  of  prior  probabilities,  in  the  equation  for  setting  the 
threshold  score,  we  now  observe  what  happens  when  the  second  component,  the  values  associated  with 
the  decisions,  are  changed.  In  this  example,  the  prior  probabilities  revert  back  to  1/2  for  each  group,  but 
now  it  is  more  important  to  make  a  correct  decision  about  a  predicted  PASS  than  about  a  predicted  FAIL. 
To  illustrate  this  numerically  let, 

P(Fail)  =  .50 
P(Pass)  =  .50 

13 


Vi  =5 


V2  =0 
Vg  =0 
V4  =1 

_  P(Fail)  F4-V3 

^  P(Pass)  Fi  -  V2 

.5  1-0 

^^5-0 

=  .20 

13  moves  to  the  left  along  the  score  axis  as  the  value  of  a  correct  decision  about  a  PASS  increases  in 
relative  value  to  a  correct  decision  about  a  FAIL.  Figure  4  shows  this  movement  of  the  threshold  score  for 
a  change  in  the  relative  evaluation  of  decisions.  If  it  is  important  to  c^ture  in  the  prediction  most  of  the 
students  who  could  pass  without  worrying  as  much  about  those  predicted  to  PASS  who  eventually  FAIL, 
then  this  movement  of  die  threshold  score  makes  sense.  The  symmetrical  situation  of  having  a  costly 
environment  for  &ilures  would  cause  the  threshold  score  to  move  to  the  right  and  make  it  more  difficult 
for  a  student  to  score  above  the  cut-ofif  score.  This  type  of  placement  for  the  threshold  score,  of  course, 
would  appear  in  higher  values  for  V4  relative  to  Vi. 


Fail  Pass 


Figure  4;  The  threshold  score  moves  to  the  left  with  a  change  in  the  evaluation  of  a  correct  predicted 
pass  relative  to  the  other  decisions. 
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The  numerical  examples  in  this  section  have  shown  the  threshold  score  is  sensitive  to  both  the  prior 
knowledge  about  success  and  failure  and  to  the  relative  values  that  are  attached  to  the  predictions  for 
success  or  failure  The  threshold  score  moves  in  a  direction  justified  by  Bayes’s  Formula  when  either  one 
or  both  of  these  components  are  changed. 


5  Decision  Theory  Implications  for  LCAC 

All  the  ingredients  involved  in  the  SDT  treatment  of  the  LCAC  prediction  algorithm  have  now 
been  presented.  As  a  summary,  the  calculation  and  figures  for  the  actual  LCAC  situation  are 
^ven  in  this  section.  Figure  5  shows  the  theoretical  normal  curve  of  the  composite  score  data 
for  the  FAIL  group  with  ufail  =  .14135  and  apAii  =  LOO  and  the  corresponding  normal  curve 
for  the  PASS  group  with  fipAss  =  .77239  and  (Tpass  =  1.00. 
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Figure  5:  The  theoretical  normal  curves  for  the  composite  scores  of  those  LCAC  candidates  who  passed 
training  and  those  who  did  not 

The  probability  density  function  (the  y-axis)  of  each  of  these  curves  goes  into  the  calculation 
of  the  likelihood  ratio,  C{x).  The  likelihood  ratio  at  the  threshold  score  is  equal  to  /?.  Frcan  the 
definition  of  >C(a;), 

P(Data  =  .14135|Pass) 

“  P(Data  =  .  14135  |Fail) 

The  threshold  score  is  located  at  the  mean  of  the  FAIL  distribution.  From  a  table  of  the  normal 
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distribution,  we  can  find  that 


P(Data  =  .14135|Fail)  =  .3989 


To  determine  P(Data  =  .14135|Pass),  we  observe  that  a  score  of  .14135  is  — .631cr  from  the 
mean  of  the  PASS  distribution.  The  normal  curve  has  a  probability  density  of  .3270  at  this 
z-score.  Therefore, 


C{x) 


.3270 

.3989 

.8198 


The  actual  historical  fi-equency  of  the  attrition  rate  before  the  selection  system  was  implemented 
was  about  40%.  Therefore,  P(FAIL)  =  .40  and  P(PASS)  =  .60,  and  we  can  then  determine  what 
the  costs  of  the  decision  must  be  for  the  threshold  score  set  at  the  mean  of  the  FAIL  distribution. 
Establishing  a  scale  by  setting  the  value  of  the  two  incorrect  decisions  V3  and  V2  at  0  and  the 
value  of  the  correct  decision  Vi  at  1,  V4  is  found  by 


P(Fail)  ^ 
P(Pass) 

1  1 

.40  14  - 

.60  ^  14  - 

14 

V2 

p  = 

.8198 

.8198  = 

.6667a: 

X  = 

1.2296 

X  = 

1  1 

II 

1  1 

1.23  -  0 

1-0 

14  = 

1.23 

By  valuing  a  correctly  predicted  FAIL  slightly  more  than  a  correctly  predicted  PASS,  we  move 
the  threshold  to  the  right.  Recall  the  discussion  of  the  effects  of  the  prior  probabilities  and  the 
decision-true  state  matrix  on  the  direction  of  movement  in  the  threshold  score.  This  is  also  the 
explanation  promised  in  the  previous  section  for  why  the  ratio 

P(Fail) 

P(Pass) 

was  ^ven  the  particular  values  in  the  numerical  exercise. 
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As  mentioned  before,  V4  was  arbitrarily  manipulated  to  bring  the  threshold  score  to  occur 
at  the  mean  of  the  P(Data|FAIL)  distribution.  In  an  ideal  world  the  prior  probabilities  and  the 
decision-true  state  matrix  would  be  known.  These  known  values  would  then  determine  /?,  and 
the  threshold  score  could  be  set.  In  the  present  scenario,  this  sequence  of  actions  was  reversed 
by  pinning  the  threshold  score  at  a  precise  location  and  then  calculating  what  this  implied  for 
the  value  of  the  correct  prediction  of  FAIL  (V4). 

As  always,  the  threshold  score  could  be  changed  by  different  knowledge  about  either  the 
prior  probabilities  of  a  PASS  or  FAIL,  or  by  a  different  set  of  values  for  the  correct  and  incorrect 
decisions.  For  example,  if  a  correct  prediction  about  a  FAIL  was  valued  even  more  highly  than 
in  the  current  example  (I4  >  1-23),  then  the  threshold  score  would  move  to  the  right.  It  would 
then  be  harder  for  a  candidate  to  pass  the  selection  battery  and  there  would  be  more  predicted 
FAILS  and  fewer  predicted  PASSES.  The  down  side  of  moving  the  threshold  in  this  manner  is 
that  there  would  also  be  more  incorrect  decisions  when  predicting  fail  for  candidates  who  would 
have  passed.  But  this  trade-off  is  implicitly  accounted  for  within  the  factors  making  up  /?.  The 
decisions  ensuing  from  a  particular  placement  of  the  threshold  are,  by  definition,  the  optimal 
decisions  given  the  prior  probabilities  and  decision-true  state  matrix. 
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